Skip to content

fix(watchdog): skip cron-liveness rescue when daily-regen is disabled - #10540

Merged
MarkusNeusinger merged 9 commits into
mainfrom
fix/watchdog-skip-disabled-daily-regen
Aug 24, 2026
Merged

fix(watchdog): skip cron-liveness rescue when daily-regen is disabled#10540
MarkusNeusinger merged 9 commits into
mainfrom
fix/watchdog-skip-disabled-daily-regen

Conversation

@MarkusNeusinger

Copy link
Copy Markdown
Owner

Problem

Watchdog: scan (cron) has been failing on 4 of its last 6 runs (2026-08-22 → 2026-08-24). The failure is always the last thing the step does:

##[warning]daily-regen: no new run on main for 133h49m (> 10h) — schedule starved, re-dispatching
##[notice]daily-regen: cron starved (133h49m silent) → dispatching
could not create workflow dispatch event: HTTP 422: Cannot trigger a 'workflow_dispatch' on a disabled workflow
##[error]Process completed with exit code 1.

daily-regen.yml is currently disabled_manually. A disabled workflow has exactly the signature the section C liveness check looks for — its cron never ticks, so the gap grows without bound — but dispatching it is impossible, and the resulting non-zero gh exit kills the step under set -euo pipefail. Sections A and B had already completed their scans by then, so the actual watchdog work happened and was then reported as a failed run.

Fix

Read the workflow's state before attempting the rescue and skip with a notice unless it is active:

REGEN_STATE=$(gh api "repos/${GH_REPO}/actions/workflows/daily-regen.yml" --jq '.state' 2>/dev/null || echo unknown)
if [[ "$REGEN_STATE" != "active" ]]; then
  echo "::notice::daily-regen liveness: workflow state is '${REGEN_STATE}', not active — rescue skipped"
elif (( HOUR >= 17 && HOUR <= 21 )); then
  ...

A deliberate pause stays paused; genuine scheduler starvation on an active workflow is still rescued exactly as before. The || echo unknown fallback means an API hiccup also skips the rescue rather than failing the scan.

Verification

  • yaml.safe_load parses the workflow.
  • The step's run: block extracted and checked with bash -n.
  • Behaviour is only observable on real scheduled runs (this is one of the documented no-verification-loop areas in CLAUDE.md), so the change is deliberately minimal: one state read, one ifelif.

Note for the reviewer

Whether daily-regen should be re-enabled is a separate, human decision — this PR only stops the watchdog from crashing on the answer being "no".

🤖 Generated with Claude Code

https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP

The liveness check in section C treats a >10 h gap in daily-regen runs as a
starved GitHub schedule and re-dispatches the workflow. A manually disabled
daily-regen produces exactly that signature — its cron never ticks, so the gap
grows without bound — but `gh workflow run` on a disabled workflow returns
HTTP 422. Under `set -euo pipefail` that killed the whole scan step after
sections A and B had already done their work, so every scan outside the quiet
window reported failure (4 of the last 6 runs).

Read the workflow's state before the rescue and skip with a notice unless it is
`active`. A deliberate pause stays paused; genuine scheduler starvation on an
active workflow is still rescued.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP
Copilot AI lite review requested due to automatic review settings August 24, 2026 08:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, matches the PR’s stated failure mode, and avoids failing the watchdog scan when daily-regen is intentionally disabled.

Pull request overview

This PR prevents the scheduled watchdog workflow from failing when it attempts to “rescue” the daily-regen cron while that workflow is manually disabled, by checking the target workflow’s state before dispatching.

Changes:

  • Add a pre-check in the watchdog cron-liveness section to read daily-regen’s Actions workflow state and skip the rescue when it isn’t active.
  • Add a [Unreleased] changelog entry describing the watchdog fix and the prior failure mode (HTTP 422 on dispatching a disabled workflow).
File summaries
File Description
CHANGELOG.md Documents the watchdog fix under [Unreleased] → Fixed.
.github/workflows/watchdog-stuck-jobs.yml Skips cron-liveness rescue dispatch when daily-regen is not active, avoiding false watchdog failures.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings August 24, 2026 20:32
@MarkusNeusinger
MarkusNeusinger enabled auto-merge (squash) August 24, 2026 20:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, correctly avoids the known HTTP 422 failure mode for disabled workflows, and preserves the existing rescue behavior for active schedules.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 24, 2026 20:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, correctly avoids dispatching a disabled workflow (preventing the observed HTTP 422 failure under set -e), and preserves the existing rescue behavior for active workflows.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 24, 2026 20:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, correctly guards the gh workflow run path when daily-regen.yml is not dispatchable, and includes a corresponding changelog entry.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 24, 2026 20:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is small, well-scoped to the failing rescue path, and preserves the existing behavior for active workflows while preventing false-negative watchdog failures.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 24, 2026 20:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, matches the PR description, and prevents a confirmed failure mode without altering the existing rescue behavior for active workflows.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 24, 2026 20:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, matches the stated failure mode, and safely avoids a known gh workflow run failure path without altering the successful watchdog scans (sections A/B).

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings August 24, 2026 20:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, well-scoped to the failing watchdog path, and safely avoids a known gh 422 failure mode while preserving the existing rescue logic for active workflows.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

…abled-daily-regen

# Conflicts:
#	CHANGELOG.md
Copilot AI review requested due to automatic review settings August 24, 2026 20:51
MarkusNeusinger added a commit that referenced this pull request Aug 24, 2026
#10628)

## Why

The `babysit-pipeline` skill covers watching one fresh spec to 15/15.
Backfilling coverage gaps across the catalogue — dozens of older specs
each missing the libraries added after they were generated — is a
different job, and the skill said nothing about it. Running it by hand
on 2026-08-24 surfaced four traps that cost, or nearly cost, real
implementations.

## What's new

**Section 5 · Gap backfill**

- Compute the missing set from `plots/{spec}/metadata/{lang}/{lib}.yaml`
on `origin/main`, **not** from `impl:{lib}:done` labels — most affected
specs have closed issues, so their labels are absent or stale. Includes
the `git ls-tree | awk` one-liner.
- `run_spec.sh <spec> <model> <lib>...` as the driver: staggered
dispatch, poll to metadata, `RESULT=COMPLETE|PARTIAL|TIMEOUT`, skips
libraries already on main so a re-run retries exactly the gaps.
- Two specs in parallel (~4 specs/h vs ~2; ten concurrent
`impl-generate` runs showed no rate-limit effects).
- A `done.log` / `deferred.log` ledger written *before* dispatching the
next spec, so a compaction or crashed session resumes without
recounting.
- **The one-retry rule**, with the evidence for why it is not optional:
the single retry recovered `highcharts/treemap-basic`,
`ggplot2/wireframe-3d-basic` and `ggplot2/network-force-directed`, all
three of which had already been read as capability gaps.

**Four new gotchas**

| Trap | Why it matters |
|---|---|
| `Marking <lib> as failed: N generation attempts` counts more than this
run | The markers span past campaigns (#10627 scopes this to 12 h).
Check the `Previous failures for <lib>/<spec>: N` notice before
concluding a library can't do a plot type. |
| `impl:<lib>:failed` is terminal **and** unreliable | Nothing
re-dispatches it — watchdog case 3 fires once, then only logs `needs
manual attention`. And of 87 such labels, **42 sat on implementations
that had since landed**. |
| "Agent reports success, writes no file" | 8 of 85 generate runs (~9%).
Self-heals when retry budget remains (`bubble-basic/highcharts`: failed
17:55, succeeded on auto-retry 18:02). One occurrence is noise. |
| Static library + interactive/3D spec | The one gap shape that usually
is genuine — 18 of the 45 real gaps. Still give the one retry, then
defer. |

**`run_spec.sh` joins the bundled scripts.** Its hardcoded
`REPO=/home/tirao/anyplot` is replaced by resolution from the script's
own location (`ANYPLOT_REPO` overrides), matching how `poll_spec.sh`
derives `HERE`.

## Verification

- `bash -n` on `run_spec.sh`; repo auto-resolution checked from the
skill directory (`git -C "$HERE" rev-parse --show-toplevel` → the repo
root).
- The skill's existing `ALL_LIBS` gotcha now names `run_spec.sh`'s
`lang_of` too, so a library addition updates both scripts.
- Content is drawn from a real backfill run, not invented: every number
above has a corresponding workflow run or label query behind it.

## Related

- #10627 fixes the retry-cap bug the first gotcha describes.
- #10540 fixes the watchdog crash found in the same session.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@MarkusNeusinger
MarkusNeusinger merged commit 477f202 into main Aug 24, 2026
7 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the fix/watchdog-skip-disabled-daily-regen branch August 24, 2026 20:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is minimal, matches the stated failure mode, and safely avoids failing the watchdog scan when the target workflow cannot be dispatched.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

MarkusNeusinger added a commit that referenced this pull request Aug 26, 2026
…er's fetch (#10660)

## Why

#10628 documented gap backfill while the backfill was still running.
Finishing it falsified three of its claims, so this PR corrects them
rather than leaving confident wrong guidance in a skill. Running the
backfill in two parallel slots also exposed a race in the driver, fixed
here.

**Scope: prose plus one executable change.** `SKILL.md` and
`CHANGELOG.md` are documentation; `run_spec.sh` gains a fetch retry
(behaviour change, details below).

## Corrections

**1. Halt-on-cluster counted the wrong thing.** The threshold was "≥5
failed `Generate:` runs in minutes = quota exhausted". Since #10627
restored the full retry budget, a single impossible pair spends three
runs on its own auto-retries — so two bad pairs trip a raw count of five
while the pipeline is perfectly healthy. That happened during the run: 6
failures at 22:29–22:37, all of them `plotnine` on two specs, nothing
wrong with the pipeline. Now counts distinct `(spec, library)` pairs.

**2. "Static library + interactive/3D spec is the one gap that is
usually real" was backwards.** Every category-level prediction made
during the backfill turned out wrong:

| Prediction | Outcome |
|---|---|
| chartjs can't do treemap / sankey | chartjs succeeded on
`network-force-directed`, `arc-basic`, and every JS-block spec |
| plotnine can't do 3D | `bar-3d-categorical` succeeded; `scatter-3d`
did not |
| ggplot2 can't do wireframe | succeeded on retry |
| pygal is a lost cause (6 failed labels) | succeeded on
`map-marker-clustered` and `line-stress-strain` |

17 of 20 parked pairs generated fine. Exactly three failed under a full
budget — plotnine on `scatter-3d`, `contour-3d`, `line-3d-trajectory`,
all of which need a spatial projection plotnine does not have, while the
"3D" spec representable in 2D went through. The gotcha now says:
measure, don't predict.

**3. The one-retry rule needed a precondition.** The workflow now spends
three attempts per campaign by itself, so a pair that comes back missing
may already be measured and the manual retry adds nothing. The skill now
shows the actual command — `gh run list` has no per-spec filter, so it
filters the output by run title with jq. Three failures minutes apart is
a gap, one is a flake. The same check resolves `RESULT=TIMEOUT` with
`recent generate failures: 0`, which only means the failures aged out of
the driver's 25-minute window (seen on `line-3d-trajectory`, which had
in fact failed three times 40 minutes earlier).

## New gotcha

Spec IDs harvested from `impl:*:failed` issue titles must be intersected
with the real `plots/` directories: **14 of 26** pointed at specs that
no longer exist on main, and such a dispatch dies seconds in at
`Validate specification exists`. This is how the rescue list first read
as 45 missing implementations when only 20 were real.

## Behaviour change: `run_spec.sh` retries its fetch

Two drivers polling the same checkout collide on the ref lock:

```
error: cannot lock ref 'refs/remotes/origin/main': is at 6b666c1 but expected c736a84
```

A lost fetch leaves `origin/main` stale, so `meta_present` understates
what has landed and the poller reports `PARTIAL` for libraries that are
already merged — a failure that disguises itself as a stalled spec.
`fetch_main()` now retries three times with backoff and, when all three
lose, logs the warning **with git's own stderr** so a ref-lock collision
is distinguishable from an auth or network failure.

## Verification

- `bash -n` on `run_spec.sh`; the failure path exercised against a bad
ref → `last error: fatal: couldn't find remote ref …`, confirming the
message survives to the log.
- Every number in the corrections traces to a workflow run or label
query from the 2026-08-24 backfill; the ledger is in
`agentic/runs/babysit-2026-08-20/`.

## Related

- #10627 (retry cap) — verified in production: `Previous failures for
plotnine/contour-3d since 2026-08-24T10:38:22Z: 2` followed by `3 failed
attempt(s) in the last 12h (cap: 3 per campaign)`.
- #10628 — the section this corrects.
- #10540 — verified by dry run: `daily-regen liveness: workflow state is
'disabled_manually', not active — rescue skipped`, scan completing
successfully.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants